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ABSTRACT 


This paper discusses self-reported emotions experienced by 
students in a Massive Open Online Course (MOOC) learning 
context. Emotions have been previously shown to be related to 
learning in classrooms and laboratory studies and have even been 
leveraged to improve learning. In this study, frequently occurring 
discrete emotions as well as frequently, co-occurring pairs of 
emotions were analyzed during learning with a MOOC. Both 
discrete and co-occurring emotions were related to students 
dropping out of the course, illustrating the importance of student 
emotion ina MOOC context. 
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1. INTRODUCTION 


Emotion is one of the key aspects of the learning process [9,22]. It 
influences learning in a variety of ways [12], both positively (e.g., 
when a student feels engaged [19]) and negatively (e.g., during 
boredom [6,19]). These connections between emotion and 
cognition can be leveraged to improve learning [10]. For example, 
a dialog-based, intelligent tutor that adjusts its dialog to address 
negative emotions can improve learning for low-knowledge 
students [11]. Indeed, the relationship between emotion and 
learning has been researched in a variety of digital learning 
contexts in both laboratory studies and classroom studies [1,5,9]. 
There are, however, additional learning contexts in which the 
relationship between emotion and learning is less clear. In this 
study we focus on the role of emotion as it relates to student 
dropout in the context of a Massive Open Online Course 
(MOOC). 


MOOCs are an online learning context that has recently become 
popular worldwide [18]. MOOCs provide education access to 
large groups of people, many of whom are often non-traditional 
students. Little is known about the relationship between emotions 
and learning in a MOOC context. Some initial work toward 
examining emotion in MOOCs indicated that some emotions were 
related to dropout [13]. However, these results were derived from 
retrospective reports of emotion after a course rather than reports 
in the moment, 1.e., during the course. Similarly, studies have 
used MOOC discussion forums and clickstream data to infer 
student emotions such as Confusion and Frustration based on 
researchers’ judgments of how these emotions are manifested 
[16,27], but there was no measurement of the emotions from the 
students themselves. 
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The current paper expands on this limited research, addressing 
key open questions about student emotions gathered from self- 
reports at different points in a MOOC. We explore a range of 
emotions, including Anger, Boredom, Confusion, Contentment, 
Disappointment, Enjoyment, Frustration, Hope, Hopelessness, 
Isolation, Pride, Relief; Sadness, and Shame, while also focusing 
on the relationship between Anxiety and learning statistics (the 
focus of the MOOC in this study) [8,17]. 


We also consider the possibility of co-occurring emotions. 
Decades ago, Izard et al. [14] considered the possibility that 
certain emotions may be experienced in concert with other 
emotions, rather than individually. Experimental research has 
shown this to be the case in some situations, for example with 
induced emotions and even with emotions experienced during 
everyday life [3,21]. In the context of learning, Bosch and 
D’Mello [4] studied novice programmers’ emotions and found 
Confusion co-occurred with Frustration, while Curiosity co- 
occurred with Engagement. The degree of co-occurrence of 
Curiosity and Engagement was positively correlated with learning 
(r = .226) after accounting for individual occurrences, thereby 
highlighting the importance of examining co-occurring emotions. 


In addition to tracking the incidence of emotions and co- 
occurrence pairs, we also consider how emotions are related to 
key educational outcomes. Early studies of MOOC data and 
student behavior [7,26], have often focused on “dropout” as both a 
problem and a key outcome. Recently, some have questioned the 
validity of dropout as a metric of outcome assessment [13]. 
However, Yang et al. [26] have noted, for instance, that the very 
low completion rates of MOOCs should signal some concern. 
Researchers have used log data to predict student dropout [15,23] 
as part of a larger effort aimed at better understanding student 
dropout from MOOCs and, in tum, improving the MOOC 
learning experience to reduce dropout. Here, we consider the 
relationship between students’ self-reported emotions and course 
dropout. 


To our knowledge, this is the first study to measure a range of 
self-reported student emotions in a MOOC context. We believe 
that the opportunity to study student emotion with large courses in 
the wild offers a valuable addition to previous work that has 
focused more on laboratory settings or traditional classroom 
environments. We address three related questions in this research: 


* QI. What emotions do students experience ina MOOC? 

* Q2. Which emotion pairs co-occur more than chance? 

¢ Q3. How do individual and co-occurring emotions relate to 
dropout? 
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2. METHOD AND COURSE SETUP 

“T Heart Stats” was an introductory Statistics MOOC offered by a 
university in the Midwestern United States. One goal of the 
course was to alleviate student anxiety towards statistics. In this 
regard it was a prime opportunity to analyze student affect in a 
MOOC setting, while also providing an opportunity to study 
student affect at scale in the wild. 


This MOOC contained eight modules covering topics ranging 
from levels of measurement to ANOVA. Modules were 
designed to be completed in sequential order. Nevertheless, all 
modules were released to students at the same time, so students 
were free to complete the modules at their own pace and in 
whatever order they desired. 


We used a “Pick-Two” list of 15 discrete emotions (Figure 1) to 
measure student affect. In addition to the typical set of learning- 
centered affective states like Confusion and Boredom [9], the list 
included several additional emotions, such as Enjoyment, Pride, 
Isolation, Hope, and Shame. These emotions were, in part, 
selected from Pekrun’s description of academic emotions [20]. 
One limitation of this emotion list was that Neutral was not 
included. Students were prompted to report emotions at the start 
of even-numbered modules (0, 2, 4, 6) as well as at the end of 
module 8 (last module). We only collected affect reports on 
every other module to minimize intrusion. 


Of the 24,279 students from 183 different countries enrolled in 
the course, 3,591 students reported exactly two emotions on at 
least one module. These 3,591 students constituted the sample in 
this study. Students were able to report greater or fewer than two 
emotions, but because we were interested in co-occurrence, we 
excluded responses that did not consist of exactly two emotions. 


2) What emotions best describe how you’re 
feeling about the course? Pick 2 emotions. 


Anger 

Anxiety 
Boredom 
Confusion 
Contentment 
Disappointment 
Enjoyment 
Frustration 
Hope 
Hopelessness 
Isolation 

Pride 

Relief 
Sadness 
Shame 


Figure 1. Emotion reporting list 


In addition to five “course-level” affect surveys, in which students 
reported their emotions in relation to the course as a whole, we 
also included seven “content-level” surveys. These content-level 
surveys were spread throughout the course and prompted students 
to report their emotions in response to different video lectures and 
problem sets. These are two common content-delivery methods 
for MOOCs, thereby providing a preliminary understanding of 
student affect when completing these two activities. 


3. RESULTS 


We used both the course-level and content-level students self- 
reported emotions to answer our research questions (see 
Introduction). 


Q1. What emotions do students experience in a 
MOOC? 


Figure 2 presents the aggregated proportions of each reported 
emotion across all five course-level surveys. We note that Hope 
and Enjoyment were the most frequently reported emotions. Other 
frequently reported emotions were Contentment, Anxiety, and 
Pride, while Shame, Disappointment, Isolation, Anger, and 
Sadness were rarely reported. 
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Figure 2. Proportions of self-reported emotions 


These results differ from the recent D’Mello meta-analysis [9], 
where the studies rarely included emotions such as Hope, 
Enjoyment, and Contentment. However, the focus there was on 
short one-on-one interactions during learning with technology. A 
different set of emotions appear to be playing a critical role in the 
MOOC context, so context clearly matters. It is, however, difficult 
to separate context differences from measurement differences in 
the present study. 


In addition to the course-level emotion surveys, we also included 
content-level affect surveys to assess self-reported emotion in 
relation to specific segments of content that may elicit different 
emotional responses. We selected 4 content-level affect surveys to 
highlight different affective states across video and problem set 
sections of content. Two of the activities were instruction videos 
and the other two were homework and practice problem sets. We 
excluded emotions that occurred in less than 1% of the responses 
for each specific activity. In addition, since all of the content for 
this course was released at the same time, we use log timestamps 
to ensure that: 1) Students engaged with the activity, 2) Students 
answered the activity-specific affect question after their 
engagement with the activity, and 3) Students did not take more 
than 1 hour following the last activity log to complete the emotion 
survey. 


Figure 3 presents the emotion proportion distributions for four 
learning activities. The results indicated that unlike the course- 
level emotion reports, Enjoyment was more frequent than Hope. 
Further, while Anxiety was the fourth most commonly reported 
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emotion at a course-level, it was far less prominent at the content 
level. 
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Figure 3. Proportion of emotion self-reports by activity type 


We also note that the content-level emotions varied with regard to 
certain activities. For instance, Pride was reported nearly 10 times 
more frequently in response to Module 4 Practice Problems than 
in Module 1 Video. Frustration, Confusion, and Anxiety were 
quite prominent during Module 2 Homework Problem compared 
with Module 4 Video. Relief, on the other hand, did not fluctuate 
substantially among these four content-level reports. Hope was 
more frequently reported in both of the video activities, while 
Pride was more frequently reported in the problem sets. Further 
research is needed to determine if indeed students expressed Pride 
more frequently in contexts of achievement such as completing a 
problem set. We would also need to consider a larger set of 
activities to establish if certain emotions occur more frequently 
and significantly among certain genres of content. 


These course-level affect surveys highlight that students 
experience different emotions during different types of content in 
a MOOC. If MOOCs are able identify the prominent emotions 
associated with various types of content such as videos and 
problem sets, then instructors and course designers can provide 
appropriate support to learners when needed. 


Q2. Which emotion pairs co-occur more than 


chance? 

Bosch and D’Mello [4] investigated co-occurrence of emotions in 
a computerized learning environment. In their study, they 
employed a retrospective judgment protocol without any 
interruptions during the learning session. They determined which 
co-occurring emotions occurred more than chance by computing 
Lift scores [24] for each emotion pair. Lift is a technique from 
association rule learning that can be used to compare the observed 
co-occurrence of emotions to the level expected by chance. Lift of 
a pair of emotions (X, Y) is defined as ratio of Pr(X and Y) to 
Pr(X)*Pr(Y). 


We identified co-occurring course-level emotions as follows. 
First, we only considered responses with exactly two emotion 


reports. Second, we only considered affective states that occurred 
at least 1% of the time. This restricted our analysis to Anxiety, 
Boredom, Confusion, Contentment, Enjoyment, Frustration, 
Hope, and Pride. Lift scores were calculated for all pairwise 
combinations of the above emotions. We used random sampling 
without replacement (1,000 iterations) and a sample size of 3,000 
to compute 95% bootstrapped confidence intervals for the Lift 
scores. Lift scores above 1.0 with confidence intervals that do not 
overlap with 1.0 are considered to occur more frequently than 
chance. 


We computed Lift scores for all 5 course-level affect reports. 
There were 92 distinct co-occurring emotions and a total of 5,189 
emotion pairs as reported by 3,591 learners. The results are shown 
in Table 1. We note that only 5 out of the possible 92 emotion 
combinations co-occurred at levels above chance and these mainly 
involved the learning-centered affective states of Confusion, 
Frustration, Boredom, and Anxiety. The Confusion + Frustration 
pair had the highest Lift score, which is consistent with [4] despite 
considerable differences in the temporal resolution of the 
analyses. Somewhat surprising is the fact that Boredom co- 
occurred with both Confusion and Frustration, but this might be 
attributed to the coarse-grained nature of the emotion self-reports 
(e.g., Boredom could occur for some activities and Confusion for 
others within the same session). 


Table 1. Lift of frequently co-occurring emotion combinations 


Emotion Pair Mean (SD) Confidence Interval 
ae 1.22 (0.17) (1.21, 1.22) 
nae. 1.06 (0.23) (1.05, 1.06) 
eon 1.39 (0.43) (1.39, 1.4) 
Confusion + 3,22 (0.41) (21, 3.23) 


Frustration 


Q3. How do individual and co-occurring 


emotions relate to dropout? 

We coded a student as having “dropped out” if he or she had no 
interaction events in the last module (Module 8). Table 2 presents 
partial Spearman’s rho between dropout and course-level discrete 
emotions that comprised at least 1% of the data and corresponding 
exceeding chance. We partialled out the number of emotion 
reports per student in order to control for the steep rate of attrition 
and subsequent dropout bias in our data. 


The results indicated that Anxiety, Confusion, and Frustration 
were significantly positively correlated with dropout, which is 
what we would expect. It was surprising, however, that Hope was 
also positively correlated with dropout, suggesting that these 
hopeful students might have become disillusioned by the MOOC. 
Relief was weakly negatively related to dropout, albeit non- 
significantly. 
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Table 2. Partial correlations between affect reports and dropout 


Emotion/ rho 
Combination P 
Anxiety 155.000 
Boredom 004 .954 
Confusion 122 = .019 
Contentment -.035 243 
Enjoyment -.028 .184 
Frustration 251 = .003 
Hope 046 =.018 
Pride 034 .476 
Relief -.081  .145 
Anxiety + Frustration 107.458 
Boredom + Confusion -.088  .684 
Boredom + Frustration -.018 .956 
Confusion + Frustration 77.263 


The most valuable payoffs of this study for learning scientists and 
MOOC designers are the positive, though weak, correlations 
between Frustration, Anxiety, Confusion and dropout. The next 
step is to identify the causes or partial causes of those negative 
emotions. For example, students reported three times more 
Frustration in Module 2 Homework Problems than in other 
selected activities, suggesting that the homework problems in this 
module might need deeper consideration. 


4. DISCUSSION 


We recorded student affect in a MOOC setting and analyzed them 
with respect to both individual emotions and co-occurring pairs. 
This study marks the first large-scale analysis of self-reported 
emotion in a MOOC context. We found that students experience a 
rather diverse set of emotions while completing a MOOC in 
comparison with previous work that has focused on lab- or in- 
class learning. Particularly interesting was the finding that Hope, 
Enjoyment, and Contentment were the most frequently reported 
emotions in the MOOC context, given that they are rare in shorter 
learning sessions studied in previous work [9]. 


We also found that some emotions fluctuate depending on MOOC 
content. This is an especially valuable finding for both 
instructional designers and researchers. From a learning design 
perspective, if we know how students are affectively reacting to 
different types of content, we can adjust the course materials 
accordingly. 


Our findings also contribute to the dropout problem in MOOCs. 
Despite researchers capacity to predict dropout [25,26], we still 
lack a robust understanding of student dropout. We identified 
specific emotions and emotion combinations that correlate with 
student dropout, yielding an affective perspective to the dropout 
problem. 


5. LIMITATIONS AND FUTURE WORK 


There are several limitations with this exploratory study. First, the 
content was released to students all at once, so they could 
complete the course in any order they desired. This limits the 
feasibility of temporal analysis of the data. Second, since this 
study was based on a live course, we could not ask students to 
self-report their affective states as frequently as in a lab setting. 


This limits use of the data for more fine-grained sequential 
analyses. 


Our analyses also point to several opportunities for future work. 
One promising avenue is sensor-free affect detection for MOOCs 
[2]. It would be valuable to model student emotion based entirely 
on clickstream data provided by edX and other online learning 
platforms. This would allow for far more frequent affect 
measurement and more timely affect intervention. If, for instance, 
we know, based on log data, that a student is Frustrated, and we 
know that Frustration correlated with dropout, we can launch 
pedagogical scaffolds to help the student manage his or her 
Frustration. 


A second opportunity for future work is to analyze changes in 
emotions across the time. There are many questions that can be 
asked along this front. How do emotions change over the duration 
of an activity, a session, or the entire course? What is the affective 
trajectory of a successful MOOC student? Further research is 
needed to map emotion trajectories over the duration of the course 
so that we better understand the relationships between emotions, 
their temporal dynamics, and educational outcomes. 
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